Standalone single-deploy Modal packages for each stack by aleks-mitov · Pull Request #12 · AssemblyAI/streaming-self-hosting-stack

aleks-mitov · 2026-08-26T12:49:21Z

Runs each self-hosted stack on Modal's serverless GPUs. Each stack is a self-contained Modal App deployed with a single modal deploy: the app defines its servers, and the API resolves its ASR and license-proxy URLs from the same App at startup, so one command brings up and wires the whole stack.

Three self-contained packages

File	App	Servers
`sync/modal/modal_app.py`	`aai-sync-u3pro`	`SyncApi` (L40S), `LicenseProxy` (CPU)
`streaming/modal/modal_app_universal_3_5_pro.py`	`aai-streaming-u3pro`	`StreamingApi` (CPU), `Asr` (L40S), `LicenseProxy` (CPU)
`streaming/modal/modal_app_english_multilang.py`	`aai-streaming-english-multilang`	`StreamingApi` (CPU), `Lb` (nginx), `AsrEnglish` (L40S), `AsrMultilang` (L40S), `LicenseProxy` (CPU)

Each directory also ships a README and a sample client (sample_sync.py / sample_streaming.py) that prints live output.

Design

Modal Servers (@app.server). The ASR runs as an autoscaling Server (min_containers=1, target_concurrency set to MAX_OPEN_STREAMS), health-gated on grpc_health_probe; WebSocket sessions are unbounded.
Single-command deploy. modal.Server.from_name(...).get_url() resolves the ASR and proxy URLs from the same App at container start, so no address has to be wired in by hand.
Encrypted ASR hop. h2_enabled=True makes Modal's TLS edge advertise ALPN h2, so the API's default-TLS gRPC client connects with AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE=True. (encrypted_ports alone negotiates no ALPN and fails with missing selected ALPN property; h2_ports / h2_enabled is required.)
Secure by default. StreamingApi / SyncApi ship unauthenticated=False (Modal proxy auth, enforced on the WebSocket upgrade); AAI_REQUIRE_MODAL_AUTH=0 opts into a throwaway public test endpoint.
Supervised binaries. Each server fate-shares its vendor process with the container, so a post-startup crash triggers a Modal replacement rather than a silent black hole.
Operator-configurable audio limits. The sync caps (MAX_AUDIO_DURATION_MS, etc.) read from the environment with the compose defaults as fallback.
Model routing. The single-model u3pro stack needs no router; the english/multilang stack keeps an nginx Lb that routes the x-model-version gRPC metadata (en-default / ml-default) to the two backends.
Image tags. streaming-api and the Universal-3.5 Pro ASR are pinned to release-v1.0.1 (the API image logs peer-aborted handshakes at WARNING rather than ERROR); the license-and-usage-proxy stays on release-v1.0.0.

Security note

The public front door (StreamingApi / SyncApi) ships with Modal proxy auth on by default (unauthenticated=False), enforced at Modal's edge including the WebSocket upgrade. Verified live: an unauthenticated POST /transcribe gets a 401 at the edge, an authenticated call returns a real 200 transcript. That is the real access gate.

The residual exposure is the two internal services. The internal hops (StreamingApi to the ASR / Lb, and to LicenseProxy) cross Modal's public TLS edge because Modal has no private inter-container network by default. The hop is encrypted (TLS h2), but the backends and proxy run unauthenticated=True since the API dials them server-side and cannot attach Modal auth headers, so their .modal.direct URLs are long and structured but public. That is obscurity, not access control. The risk is bounded: forging a POST /v1/usage inflates the self-hoster's own usage bill (it does not grant free service), and reaching the ASR directly needs the URL and buys time on the operator's own GPU. No audio leaves the deployment and there is no cross-tenant surface.

Taking the internal services fully private is a Modal limitation for this deployment shape, not a code gap. The two ways to do it both fall down here: co-locating everything in one container is not viable (three separate vendor images), and Modal's i6pn private fabric only connects containers in the same underlying datacenter, which the credit-card plan does not guarantee (the L40S GPU ASR is placed in a different datacenter than the CPU front-ends, and i6pn does not bridge datacenters; forcing same-datacenter placement is gated behind Modal sales). The full private i6pn build was implemented and tested end to end to confirm this; it connects only when the containers happen to co-locate. See the exposure comment on this PR for the detail.

Net: a gated front door plus encrypted hops is the right posture for evaluation and POC. Closing the internal exposure for a stricter production bar needs Modal to enable same-datacenter placement.

Validated live (Modal, L40S)

Deployed from scratch and torn down:

sync: 60s WAV to 152 word-level timestamps, ~2.0s server-side.
streaming u3pro: real turns over the TLS h2 ASR hop; first turn ~1.4s warm.
streaming english + multilang: both universal-streaming-english and universal-streaming-multilingual return real turns, each routed by the nginx Lb to its own L40S backend.
License gate: both APIs log Proxy is connected and license is valid and SIGKILL on failure; an invalid license makes the proxy exit 1 at startup.
Usage tracking: every stack POSTs to usage-tracker.assemblyai.com/v1/usage with status_code=200.
Handshake logging: on the v1.0.1 API, opening handshake failed (from L4 health probes and aborted clients) logs at WARNING, not ERROR.

🤖 Generated with Claude Code

Repackage the Modal deployment so each self-hosted stack is a self-contained Modal App deployed with one `modal deploy` — no start_asr step, no modal.Dict address handshake, no cross-package dependency. Uses Modal Servers (@app.server) with sibling URLs resolved via Server.from_name at startup. - sync/modal_app.py -> aai-sync-u3pro - streaming/modal_app_universal_3_5_pro.py -> aai-streaming-u3pro - streaming/modal_app_english_multilang.py -> aai-streaming-english-multilang Fixes carried over from the #11 review: autoscaling ASR Server instead of a 24h-capped Sandbox; no 1h WebSocket session ceiling; encrypted ASR hop via h2_enabled + AAI_USE_SECURE_CHANNEL_TO_ASR_SERVICE (encrypted_ports alone negotiates no ALPN); unauthenticated=False by default (Modal proxy auth); fate-shared vendor processes; env-overridable sync audio limits. Verified end to end on L40S (single deploy each): sync 152-word transcript, autoscaled to 181x realtime at concurrency 8; streaming u3pro real turns over the TLS h2 ASR hop; english+multilang routing both models through the nginx Lb. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

sample_sync.py (POST /transcribe), sample_streaming.py (live realtime turns, model selection for the u3pro and english/multilang stacks), and a README. Both support --concurrency / --load for a quick input-load sweep and Modal proxy-auth headers. Verified live against all three deployed stacks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

The turn counter was appended only inside the live-printing branch, so --load sessions always reported 0 turns even though turns arrived (first-turn latency was correct). Track end_of_turn regardless of live; gate only printing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

…eaming_modal_stack/ Consolidate each stack's Modal artifacts into a self-contained top-level dir: sync_modal_stack/ modal_app.py, sample_sync.py, README.md streaming_modal_stack/ modal_app_universal_3_5_pro.py, modal_app_english_multilang.py, sample_streaming.py, README.md Each new dir carries its own deploy/verify/auth/teardown README. The compose READMEs (sync/, streaming/) keep a one-line pointer instead of the full Modal section, and the root README points at the two new dirs. No code behavior changes; comment cross-references updated to the new paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

The self-hosted-streaming-api v1.0.1 image carries the handshake-logging fix (DeepLearning #19523: peer-aborted WebSocket handshakes log at WARNING, not ERROR). Bump streaming-api and self-hosted-streaming-asr-universal-3-5-pro to release-v1.0.1 in the u3pro stack via per-image tags; the license-and-usage-proxy has no v1.0.1 and stays on v1.0.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

aleks-mitov · 2026-08-26T18:50:48Z

Fresh deploy + verification (v1.0.1 bump)

Tore down all three apps and redeployed from scratch off this branch in the dlange Modal workspace (L40S), then verified:

sync — 152-word transcript, ~2.0s server-side, 25.6x realtime.
streaming english/multilingual — both universal-streaming-english (8 turns) and universal-streaming-multilingual (14 turns) stream correctly, each routed by the nginx Lb to its own backend.
streaming u3pro (v1.0.1 images) — self-hosted-streaming-api + self-hosted-streaming-asr-universal-3-5-pro pulled at release-v1.0.1; streams end to end (12 turns, first turn ~1.5s).
Handshake log-noise fix confirmed in the v1.0.1 API: after firing peer-abort probes, all 42 opening handshake failed lines on the deployed streaming_api are [warning], zero [error] — i.e. DeepLearning #19523 (commit 415137e) is live in the v1.0.1 image.

Deploys used the reorganized layout (sync_modal_stack/, streaming_modal_stack/) and the relocated sample scripts. All apps left running for now.

Note: the english/multilang stack still uses self-hosted-streaming-api:release-v1.0.0, so it does not yet carry the handshake fix — bump it to v1.0.1 too if desired (only the u3pro stack was requested).

Adversarial review of PR #12 (8 findings). Fixes: - [major] english/multilang ASR MAX_OPEN_STREAMS + target_concurrency 32 -> 48 (compose parity; avoids a ~50% GPU over-provision at scale). - [major] streaming README Verify pointed at the bundled example client, which cannot send Modal proxy-auth headers and 401s on the default deploy; point it at sample_streaming.py with --modal-key/--modal-secret and note the caveat. - [minor] sync/streaming README Verify curls now send Modal-Key/Modal-Secret (they 401'd on the default proxy-auth deploy). - [minor] nginx Lb -> ASR hop now verifies the backend cert (grpc_ssl_verify on + ca-certificates trusted store), not just encrypts. - [minor] fate-share reaper no longer reports a clean shutdown as a crash: a module-level _stopping event, set by @modal.exit stop(), distinguishes an intentional teardown from an unexpected vendor exit (all three apps). - [minor] sample_streaming.py no longer crashes formatting first_turn_s when a session yields no word-bearing turns (prints n/a). - [minor/question] english/multilang now pins streaming-api via its own API_TAG. streaming-api bumped to release-v1.0.1 everywhere it is referenced (u3pro + english/multilang Modal stacks, streaming/.env.example, root README) so both streaming stacks carry the handshake-logging fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

…only for verification) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

…ectivity) Verifying the backend cert (finding 4) fails against Modal's edge: nginx cannot build the chain and errors "unable to get local issuer certificate", marking the ASR backend down so every session 3005s (confirmed live). Revert to encrypt-only (grpc_ssl_server_name on) and document the residual limitation in-place, the finding's sanctioned alternative. The backend is unauthenticated by design; co-location or i6pn is the real fix, per the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

aleks-mitov · 2026-08-26T19:23:20Z

Adversarial review fixes + streaming-api v1.0.1 everywhere — applied & verified

Ran an adversarial review (14 raw findings → 3 refuted → 8 confirmed) and addressed all of them:

[major] english/multilang ASR concurrency raised 32 → 48 (MAX_OPEN_STREAMS + target_concurrency) to match compose; avoids a ~50% GPU over-provision.
[major] streaming README Verify now uses sample_streaming.py with --modal-key/--modal-secret (the bundled example client can't send Modal proxy auth and 401s on the default deploy).
[minor] README Verify curls now send Modal-Key/Modal-Secret.
[minor] fate-share reaper no longer reports a clean shutdown as a crash — a _stopping event set by @modal.exit distinguishes intentional teardown from an unexpected vendor exit (all three apps).
[minor] sample_streaming.py prints n/a instead of crashing when a session yields no turns.
[minor/question] english/multilang now pins streaming-api via its own API_TAG.
streaming-api → release-v1.0.1 everywhere (both Modal streaming stacks, streaming/.env.example, root README) so both stacks carry the handshake-logging fix.

[minor] LB→ASR cert verification (grpc_ssl_verify): attempted, then reverted. Turning it on broke the hop — nginx can't build a chain to Modal's edge cert (unable to get local issuer certificate), marking the ASR backend down so every session 3005d (caught in live verification). Reverted to encrypt-only and documented the residual limitation in-place (the backend is unauthenticated by design; co-location or i6pn is the real fix).

Re-verified from scratch (dlange workspace, L40S): sync (26.9x realtime), u3pro v1.0.1 (12 turns), english + multilingual with the 48-cap and encrypt-only LB (8 / 14 turns). Handshake fix confirmed on both v1.0.1 streaming APIs — u3pro 42/42 and english/multilang 33/33 opening handshake failed lines at [warning], zero [error].

bgotthold-aai · 2026-08-26T20:35:31Z

nice! From organization perspective can we add folders like streaming/modal streaming/docker streaming/sagemaker etc? I can see us adding more variations here and I dont think top level makes sense

Per review feedback, nest deployment variants under each service dir instead of top-level *_modal_stack dirs, so more targets (e.g. sagemaker) slot in cleanly: sync/docker/ (compose stack, moved from sync/) sync/modal/ (moved from sync_modal_stack/) streaming/docker/ (compose stack + nginx, moved from streaming/) streaming/modal/ (moved from streaming_modal_stack/) Relative links and paths updated throughout (root README layout + pointers, docker READMEs' ../README.md -> ../../README.md and modal pointers, modal READMEs' example paths -> ../docker/example, sample-script docstrings, and the cross-file comment references). No code behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1

aleks-mitov · 2026-08-26T21:39:29Z

Good call — done. Reorganized so each service groups its deployment targets, and the top-level *_modal_stack/ dirs are gone:

sync/
  docker/   # compose stack (moved from sync/)
  modal/    # serverless-GPU package (was sync_modal_stack/)
streaming/
  docker/   # compose stack + nginx (moved from streaming/)
  modal/    # serverless-GPU packages (was streaming_modal_stack/)

I moved the existing compose stacks into docker/ too (not just the new modal/ dirs), so a future sagemaker/ etc. slots in the same way. Relative links/paths were updated throughout (root README layout + pointers, the docker READMEs' ../README.md → ../../README.md, the modal READMEs' example paths → ../docker/example, and the sample-script docstrings); READMEs' links all resolve and the stacks still deploy + verify clean.

bgotthold-aai

Approving this because I think it looks good overall.

Should we still add a .env for model to get the registry and tag out of code?

aleks-mitov · 2026-08-27T11:21:14Z

Update on the internal-hop exposure: i6pn is blocked on this plan (datacenter placement)

Earlier in this thread I reported a CPU spike suggesting i6pn (Modal's private IPv6 network) could take the ASR and license/usage proxy off the public edge. I took that all the way to a full private implementation and ran it end to end, and I need to correct the earlier optimism: it connects only when the containers land in the same datacenter, and the credit-card plan gives no way to force that.

What I built. ASR + proxy as @app.cls(i6pn=True) with no web endpoint (so no public URL at all), and the public front door moved to a @modal.web_server method on an ordinary cls. A flash @app.server sits behind Modal's stateless reverse proxy and is not on the cls i6pn fabric, so it cannot dial the backends privately; the web-server-cls form keeps it as ordinary compute. When the containers co-locate it works: the API reached the proxy over i6pn with no public URL and delivered usage to the tracker (200).

Why it can't be made reliable here (root-caused with a TCP reachability probe written to a modal.Dict): the L40S GPU ASR is scheduled into a different underlying datacenter than the CPU front-ends. Its private address comes up on one datacenter's prefix (fdaa:a65:…) while the API and proxy are on another (fdaa:a75:…), and i6pn does not bridge datacenters. The probe was unambiguous: 12 / 12 connects to the ASR timed out against one stable address, while the proxy answered in 2 ms. It is not staleness or slow propagation (the address was stable), and it is not GPU-vs-CPU (a cheap L4 on the front-ends still lands on a75, not the L40S's a65). The only lever, pinning everything into one underlying region, is rejected on this plan: only the coarse us-east group is accepted, and granular strings (us-east-1, …) are gated behind Modal sales. Co-locating all three in one container is the other route and is not viable (three separate vendor images; the ASR image has no apt to fold the others in).

Net. This is a Modal limitation for this deployment shape, not a code gap. #12 ships public-edge as evaluation-ready, with the front door authenticated (proxy auth on by default, verified 401 unauth / 200 authed) and the hops TLS-encrypted as the mitigation. Fully taking the internal services private would need Modal to enable same-datacenter placement, and I have the a65-vs-a75 evidence to make that ask concrete. The full private i6pn implementation is kept as the reference for when that lands.

The internal-hop section presented co-location and i6pn as available fixes and called the topology evaluation-suitable without qualification. Reframe to match what the i6pn investigation found: the authenticated front door is the real gate, the residual backend exposure is bounded, and taking the backends fully private is a Modal placement limitation for this shape (GPU and CPU front-ends land in different datacenters; i6pn does not bridge them; same-datacenter placement is gated behind Modal sales). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL3rQMpGXvmMHRJiffWMF1